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The  Kent  Center  is  pleased  to  publish  as  part  of  its  Occasional  Papers  series  this 
“think  piece”  on  how  to  understand  and  address  the  unique  analytical  challenges 
posed  by  complex  and  fast-moving  transnational  threats  such  as  terrorism,  WMD 
proliferation,  and  organized  crime.  Distilled  from  a  longer  piece  also  co-authored 
by  Warren  Fishbein  of  the  Kent  Center’s  Global  Futures  Partnership  and  Gregory 
Treverton  of  the  RAND  Corporation,  this  paper  proposes  some  practical  ideas  for 
adapting  the  organizational  culture  and  processes  in  which  analysis  of  these  issues 
is  done  to  improve  understanding  and  warning. 

The  authors  use  as  a  springboard  for  their  discussion  the  ideas  generated  by  a 
series  of  unclassified,  multidisciplinary  workshops  with  outside  experts  convened 
by  GFP  and  RAND  during  2003  to  explore  “Developing  Alternative  Analysis  for 
Transnational  Issues.”  (Reports  of  these  workshops  are  published  separately  by 
RAND  Corporation  in  report  CF-200.)  In  this  paper,  workshop  insights  are  coupled 
with  findings  from  further  research  on  concepts  such  as  intuitive  thinking,  sense¬ 
making,  and  mindfulness  to  suggest  an  approach  for  applying  what  the  authors  call 
“alternative  sense-making”  to  complex  transnational  issues. 

The  ideas  suggested  here,  however,  are  less  a  prescription  for  analytical  practice 
than  an  invitation  to  dialogue,  debate,  and  further  research  that  will  help  advance 
the  doctrine  of  analysis  for  transnational  threats.  The  Kent  Center  welcomes  this 
contribution  to  the  literature  on  intelligence  analysis  and  looks  forward  to  continued 
exploration  of  the  arguments  presented  here  and  in  the  longer  version,  “Making 
Sense  of  Transnational  Threats,”  published  in  Kent  Center  Occasional  Papers, 
Volume  3,  Number  1 . 


The  Director 
Sherman  Kent  Center 
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Summary 


Understanding  complex  transnational  issues,  such  as  terrorism  and  weapons 
proliferation,  requires  an  alternative  analysis  approach  that  is  more  an  ongoing 
organizational  process  aimed  at  promoting  “mindfulness” — continuous  wariness  of 
analytic  failure — than  a  set  of  tools  that  analysts  are  encouraged  to  employ  when 
needed.  This  means  that  Intelligence  Community  analytic  organizations  need  to 
institutionalize  sustained,  collaborative  efforts  by  analysts  to  question  their  judgments 
and  underlying  assumptions,  employing  both  critical  and  creative  modes  of  thought. 
For  this  approach  to  be  effective,  significant  changes  in  the  cultures  and  business 
processes  of  analytic  organizations  will  be  required. 

These  are  the  key  conclusions  arising  from  a  project  undertaken  by  the  CIA’s  Global 
Futures  Partnership  in  the  Sherman  Kent  School  for  Intelligence  Analysis  and  the 
RAND  Corporation  to  rethink  “alternative  analysis” — tools  designed  to  help  analysts 
and  decision-makers  employ  rigorous  self-review,  question  judgments,  and  explore 
alternative  outcomes — to  better  address  threats  in  the  increasingly  important  realm 
of  transnational  issues.  In  a  series  of  unclassified  workshops,  Intelligence  Community 
analysts  and  analytic  managers  came  together  on  a  non-attribution  basis  with 
outside  thinkers  in  a  broad  range  of  fields  relevant  to  the  analytic  process,  including 
cognitive  psychology,  psychiatry,  organizational  behavior,  artificial  intelligence, 
knowledge  management,  intelligence  studies,  and  the  foreign  policy  process.  Through 
presentations  and  discussions  among  participants,  the  workshops  sought  to  generate 
broad  concepts  about  adapting  alternative  analysis  to  enhance  warning  of  out-of- 
the-ordinary  actions  undertaken  by  non-state  actors,  epitomized  in  the  September 
1 1  attacks.  What  follows  in  this  report  are  some  of  the  more  intriguing  practical  ideas 
that  surfaced  at  the  workshops,  augmented  with  insights  from  related  studies  that 
elaborate  these  ideas.1 


1This  paper  is  an  abbreviated  version  of  a  project  report  entitled  Making  Sense  of  Transnational  Threats ,  Kent 
Center  Occasional  Paper,  Vol.  3,  No.  1 .,  available  on  the  CIA  public  website  at  www.cia.gov.  Summaries  of  the 
discussions  at  each  of  the  project’s  four  workshops  can  be  found  in  the  RAND  Corporation  report  (CF-200) 
Enhancing  Warning  for  Transnational  Threats:  Workshop  Reports. 
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The  Nature  and  Role  of  Alternative 
Analysis 

Traditional  intelligence  analysis 
generates  forecasts  or  explanations 
based  on  logical  processing  of  available 
evidence,  whereas  alternative  analysis 
seeks  to  help  analysts  and  policy-makers 
stretch  their  thinking  through  structured 
techniques  that  challenge  underlying 
assumptions  and  broaden  the  range  of 
possible  outcomes  considered.  Properly 
applied,  it  serves  as  a  hedge  against  the 
natural  tendencies  of  analysts — like  all 
human  beings — to  perceive  information 
selectively  through  the  lens  of 
preconceptions,  to  search  too  narrowly 
for  facts  that  would  confirm  rather  than 
discredit  existing  hypotheses,  and  to 
be  unduly  influenced  by  premature 
consensus  within  analytic  groups  close 
at  hand.  Alternative  analysis  involves 
a  fairly  intensive,  though  time  limited, 
effort  to  challenge  assumptions  or  to 
identify  alternative  outcomes,  depending 
on  the  technique  employed,  with  the 
results  captured,  implicitly  or  explicitly, 
in  a  written  product  delivered  to  relevant 
policy-makers.2 


Moreover,  alternative  analysis  may  be 
less  effective  at  enhancing  the  warning 
process  for  threats  in  the  transnational 
realm.  Alternative  analysis  involves  the 
application  of  structured  argumentation 
to  address  discrete  questions — what  if 
Ruritania  acquires  “the  Bomb?’’  But  for 
amorphous,  continuous  threats,  such 
as  terrorism  and  weapons  proliferation, 
the  specificity  and  product-driven  focus 
of  alternative  analysis  can  be  more 
problematic. 


. . .  alternative 
analysis  may  be 
less  effective  at 
enhancing  the 
warning  process 
for  threats  in  the 
transnational 
realm. 


Analysis  of  any  kind  involves  breaking 
down  problems  into  constituent  parts, 
such  as  causes  and  effects,  and  using 
logical  operations  to  identify  and 
test  hypotheses  for  the  purposes  of 
explanation  and  prediction. Traditional 
analysis  is  relevant  for  bounded 
problems — such  as  state-to-state 
diplomacy  or  internal  politics — in  which 
there  is  a  relatively  restricted  range 
of  outcomes  or  hypotheses  to  be 
evaluated.  Alternative  analysis  tools  can 
support  the  overall  analytic  process  by 
ensuring  that  more  than  one  explanatory 
hypothesis  or  projected  outcome  is 
seriously  considered. 


In  theory,  use  of  alternative  analysis 
techniques  can  help  to  reduce  the 
likelihood  of  “intelligence  failures,”  which 
historically  have  stemmed  in  part  from 
such  mental  errors  (e.g.the  ingrained 
belief  that  the  Japanese  could  not 
mount  a  successful  attack  against  Pearl 
Harbor).  In  reality,  however,  alternative 
analysis  has  not  been  particularly 
effective  within  the  Intelligence 
Community  as  it  has  been  employed 
only  sporadically,  at  best,  and  more  often 
than  not  as  a  “nice  to  have”  supplement 
tacked  on  to  traditional  analysis  rather 
than  integrated  at  the  outset  as  an 
essential  component  of  the  analytic 
enterprise  in  a  world  of  uncertainty  and 
deception. 


Intelligence  problems  in  the  transnational 
realm  are,  however,  generally  less 
bounded  than  are  those  in  the  state-to- 
state  realm.  Transnational  groups,  like 
terrorist  cells  or  proliferation  networks, 
are  likely  to  be  smaller  and  more 
numerous  than  states,  less  constrained 
by  rules  and  historic  precedent,  and 
more  affected  by  tactical  and  situational 
circumstances  as  opposed  to  deeper 
internal  drivers.  They  are  thus  more 
difficult  to  understand  than  state  actors 
and  more  capable  of  engaging  in  a  wide 
array  of  unpredictable  behaviors.  The 
transnational  arena  also  arguably  is 
more  affected  by  “information  overload" 
than  many  facets  of  the  state-to-state 
arena,  with  analysts  having  to  contend 


Alternative  analysis  includes  techniques  to  challenge  analytic  assumptions  (e.g.  “devil's  advocacy”),  and  those 
to  expand  the  range  of  possible  outcomes  considered  (e.g.  “what-if  analysis,”  and  “alternative  scenarios.” 
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Sense-making 
involves  the 
collective 
application 
of  individual 
“intuition”. . .  to 
identify  changes 
in  existing 
patterns  or  the 
emergence  of 
new  patterns. 


with  enormous  volumes  of  questionable 
data  about  their  shadowy  targets  and 
little  historical  context  for  evaluating 
such  data.  Transnational  problems 
are,  thus,  often  not  very  susceptible  to 
“analysis,’  whether  of  the  traditional  or 
alternative  variety,  because  there  are 
too  many  hypotheses  to  consider  and 
too  little  solid  information  to  support 
logical  judgments.3 

Analyzing  Complex  Issues:  The 
Concept  of  Sense-Making 


How  then  can  intelligence  address  such 
problems  and  what  are  the  appropriate 
“alternative”  processes  for  tackling 
mental  biases?  One  potentially  relevant 
concept,  developed  by  management 
scientists  to  describe  how  business  and 
public  sector  organizations  cope  with 
uncertainty,  is  the  process  of  “sense¬ 
making.”  This  is  a  continuous,  iterative, 
largely  informal  effort  by  members  of 
organizations  to  understand,  or  “make 
sense”  of  what  is  going  on  in  the 
external  environment  that  is  relevant  to 
their  goals  and  needs. 

Sense-making  involves  the  collective 
application  of  individual  “intuition” — 
experience-based,  sub-consciously 
processed  judgment  and  imagination — 
to  identify  changes  in  existing  patterns 
or  the  emergence  of  new  patterns.4 
Several  leading  management  and 
cognitive  scientists  argue,  based 
on  empirical  research,  that  intuitive 
judgment,  rather  than  formal  analysis, 


actually  underpins  most  organizational 
decisions  and  that  it  provides  at  least 
equally  reliable  results  in  predicting 
outcomes  in  most  cases.  Moreover, 
they  argue  that  it  is  superior  to  analysis 
for  problems  marked  by  high  ambiguity 
or  uncertainty,  because  efforts  to 
“reduce”  such  problems  (to  identify 
a  handful  of  key  variables)  to  fit  into 
structured  analytic  frameworks  will 
produce  misleading  results.5  Sense¬ 
making  aggregates  and  refines  intuitive 
judgment  through  conversations  within 
the  organization  in  which  members 
construct  interpretations  of  reality  and 
develop  explanations  or  stories  to 
account  for  perceived  anomalies. 

Intelligence  analysts  already  engage  in 
something  approaching  sense-making: 
for  instance,  in  the  more  thoughtful 
aspects  of  the  current  intelligence 
process.  These  involve  ongoing  efforts 
to  update  the  story  line  on  an  issue, 
usually  without  recourse  to  formal 
analytic  techniques.  Even  many 
longer-term  assessments  are  more 
accurately  described  as  intuitive  rather 
than  analytical,  if  the  latter  concept  is 
strictly  defined  as  involving  rigorous 
comparison  of  competing  hypotheses.6 
But  intelligence  “analysis”  differs  from 
sense-making  in  some  key  respects: 
it  is  generally  less  collective  (primary 
responsibility  for  sense-making  often 
resides  with  an  individual  analyst) 
and  more  formal  (with  ideas  shared 
through  the  often-adversarial  process 
of  coordination  rather  than  through 
conversation). 


3The  same  can  be  said,  however,  of  many  aspects  of  the  state-to-state  arena  such  as  state  failure,  battlefield 
dynamics,  and  crisis  diplomacy,  which  are  also  marked  by  multiple  potential  outcomes,  interactivity,  and  extreme 
information  overload. 

4See  Karl  Weick,  Sensemaking  in  Organizations,  (London:  Sage  Publications,  1995). 

5 

Gary  Klein,  Intuition  at  Work:  Why  Developing  Your  Gut  Instinct  Will  Make  You  Better  at  What  You  Do,  (New  York: 
Doubleday,  2002);  Thomas  Stewart,  "How  to  Think  with  Your  Gut,” in  Business  2.0,  November  2002, 
http://www.business2.0.com  Guv  Claxton,  Hare  Brain,  Tortoise  Mind:  How  Intelligence  Increases  When  You  Think 
Less,  (New  York:  Harper  Collins,  2000),  pp.  3-11. 

'’Richards  S.  Heuer,  Jr.,  The  Psychology  of  Intelligence  Analysis  (Washington:  Center  for  the  Study  of  Intelligence, 
1999).  See  also  Rob  Johnston,  The  Culture  of  Analytic  Tradecraft:  An  Ethnography  of  the  Intelligence  Community 
(Washington:  Center  for  the  Study  of  Intelligence,  Central  Intelligence  Agency),  in  press. 
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A  more  effective  system  of  intuition- 
based  intelligence  sense-making  offers 
a  way  to  comprehend  those  complex 
transnational  threats  whose  speed, 
uncertainty,  and  interactivity  defy 
traditional  analytic  approaches.  One 
presenter  at  the  GFP-RAND  workshops, 
a  noted  expert  on  intelligence,  made 
this  point  in  observing  that,  in  contrast 
to  traditional  Soviet-threat  type  issues, 
the  transnational  arena  (terrorism  in 
particular)  benefits  far  more  from  lots 
of  “pairs  of  eyes”  looking  at  data  for 
emerging  signs  of  threat  than  from 
detailed  analyses  of  narrowly  drawn 
issues.  While  lots  of  eyes  would,  of 
course,  be  beneficial  in  addressing 
traditional  issues,  they  arguably 
are  crucial  to  success  in  following 
transnational  issues,  where  having 
multiple  intuitions  wrestling  with  and 
sharing  perspectives  on  enormous 
information  flows  should  significantly 
improve  the  odds  in  favor  of  identifying 
threat  patterns. 

Addressing  Mental  Biases  in 
Sense-Making 

Like  traditional  analysis,  intelligence 
sense-making  suffers  from  the  effects 
of  mental  biases.  The  process  of 
challenging  assumptions  in  this  domain 
will  be  different,  however,  from  that  used 
in  the  more  familiar  analytic  domain. 
Whereas  alternative  analysis  enhances 
a  formal,  periodic  process  for  exploring 
a  limited  range  of  outcomes,  “alternative 
sense-making”  must  address  a  process 
that  is  informal,  continuous,  and 
focused  on  issues  marked  by  very  high 
uncertainty. 


A  useful  starting  point  for  identifying 
an  alternative  analysis  approach 
suitable  for  sense-making  is  the  work 
of  noted  organizational  theorists  Karl 
Weick  and  Kathleen  Sutcliffe.* 7  The 
authors  examined  a  number  of 
organizations,  such  as  nuclear  power 
plants  and  aircraft  carriers,  which  face 
challenges  in  addressing  uncertainty 
akin  to  those  faced  by  intelligence 
organizations.  Those  organizations 
on  a  daily  basis  confront  continually 
shifting  conditions  that  can  give  rise  to 
unexpected  outcomes,  with  the  potential 
for  catastrophic  consequences.8 
Nonetheless,  carriers  experience 
exceptionally  low  rates  of  error  and 
accident  in  comparison  to  other  fields 
that  also  continually  confront  uncertainty, 
such  as  medicine.  The  same  can  be 
said  of  nuclear  power  plants  where, 
according  to  another  writer,  “there  is 
no  ‘regular’  functioning  of  the  plant — 
the  status  of  the  plant  is  continually 
changing  and  therefore... anomalies  are 
very  difficult  to  track.”9 

In  studying  these  types  of  “high 
reliability  organizations”  (HROs),  Weick 
and  Sutcliffe  have  identified  lessons  that 
can  be  applied  generally  to  addressing 
uncertainty.  The  unifying  trait  of  HROs 
is  that  they  exhibit  the  quality  of 
“mindfulness,”  defined  as: 

“...the  combination  of  ongoing 
scrutiny  of  existing  expectations, 
continuous  refinement  and 
differentiation  of  expectations  based 
on  new  experiences,  willingness  and 
capability  to  invent  new  expectations 
that  make  sense  of  unprecedented 


. . .  intuition- 
based 
intelligence 
sense-making 
offers  a  way  to 
comprehend 
those  complex 
transnational 
threats 
whose  speed, 
uncertainty,  and 
interactivity 
defy  traditional 
analytic 
approaches. 


'See  Managing  the  Unexpected:  Assuring  High  Performance  in  and  Age  of  Uncertainty,  (San  Francisco:  Jossey- 

Bass,  2001). 

8Weick  and  Sutcliffe  note  that  an  aircraft  carrier  deck,  for  instance,  combines  a  volatile  mixture  of  jet  fuel,  weaponry, 
“controlled  crash”  landings,  shifting  sea  and  weather  conditions,  noise,  and  young  and  often  inexperienced 
personnel. 

9Klein,  pp.  132-134. 
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mindfulness . . . 
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events,  ...and  identification  of  new 
dimensions  of  context  that  improve 
foresight  and  current  functioning.”10 

Mindfulness  is  the  result  of  a  never- 
ending  effort  to  challenge  expectations 
and  to  consider  alternative  possibilities. 
Such  eternal  vigilance  helps  eliminate 
“blind  spots”  that  result  in  organizations 
missing  early  warning  signs  of 
unexpected  and  unwanted  change.  A 
mindful  orientation  “redirects  attention 
from  the  expected  to  the  irrelevant,  from 
the  confirming  to  the  disconfirming, 
...from  the  more  certain  to  the  less 
certain,  ...and  from  the  consensual  to 
the  contested.” 

Weick  and  Sutcliffe  identify  several 
attributes  of  organizational  culture  that 
contribute  to  mindfulness.  Two  that 
appear  to  be  critical  to  anticipating 
uncertainty,  as  opposed  to  reacting 
to  it,  are  most  relevant  to  intelligence 
organizations: 

•  A  preoccupation  with  failure, 
both  past  and  potential. 

•  A  “refusal  to  simplify.” 

High  reliability  organizations  “tend  to 
view  any  failure,  no  matter  how  small, 
as  a  window  on  the  system  as  a  whole.” 
This  preoccupation  also  extends  to 
thinking  about  future  downsides:  HRO’s 
tend  to  be  “skeptical,  wary,  suspicious 
of  quiet  periods,”  because  they  know 
that  “not  all  failure  modes  have. .  .been 
experienced  or  exhaustively  deduced.” 
Underpinning  these  attitudes  is  a 
“learning  culture"  in  which  it  is  safe 
and  even  valued  for  members  of  the 
organization  to  admit  error  and  raise 
doubts. 


The  refusal  of  HROs  to  simplify  involves 
a  refusal  to  take  things  for  granted  or 
to  rely  on  standard  interpretations. 
Personnel  in  HROs  are  relentless  in 
their  efforts  to  try  to  understand  the 
complexities  of  the  situations  they  face. 
As  an  example,  mechanics  in  nuclear 
power  plants  almost  never  rely  on  the 
simplifications  of  blueprints  when  they 
have  to  intervene  in  the  system,  but 
rather  they  personally  “walk  the  system” 
to  assure  that  no  subsequent  changes 
have  been  made  that  could  affect  the 
outcome.  Particularly  relevant  to  the 
intelligence  environment,  different 
departments  constantly  interact  when 
confronted  with  a  problem,  generating 
hypotheses  about  “what  is  going  on, 
what  can  be  done,  and  what  the  long¬ 
term,  system-wide  consequences  of 
the  proposed  action  might  be.”  Such 
interdepartmental  interaction  brings 
different  perspectives  to  the  table, 
building  a  progressively  more  complex 
vision  of  the  problem  at  hand,  and 
reducing  the  chances  that  key  aspects 
of  a  problem  will  be  overlooked. 

Applying  Alternative  Sense¬ 
making  to  Transnational  Issues 

The  concept  of  mindfulness,  as 
practiced  by  HROs,  provides  a  construct 
for  designing  processes  in  intelligence 
practice  to  improve  understanding  and 
warning  for  complex,  transnational 
issues.  For  threats  that  can  suddenly 
emerge  at  any  time,  anywhere,  and 
in  a  variety  of  forms,  analysts  need 
to  think  more  in  terms  of  a  broad 
mental  readiness  to  perceive  early 
warning  signs  of  threat  than  in  terms 
of  challenging  specific  assumptions  or 
identifying  specific  alternative  outcomes. 


10This  and  the  following  quotes  from  Weick  and  Sutcliffe  are  found  on  pages  25-62. 
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The  following  principles  can  guide  the 
development  of  an  alternative  process 
for  intelligence  sense-making.  Such  a 
process  must  be: 

•  Continual 

•  Creative 

•  Collaborative 

•  Counter-intuitive 

•  Consumer-friendly 

Continual:  The  conventional  model  for 
employing  alternative  analysis — identify 
an  issue  too  important  to  “afford 
getting  it  wrong”  and  then  challenge 
assumptions  and  identify  alternative 
outcomes — is  not  really  suitable  for 
ongoing  complexities.  There  are  too 
many  outcomes  to  be  considered,  too 
much  potential  for  sudden  change,  and 
too  many  contingent  interactions  for 
any  “one-off”  effort  to  be  particularly 
useful.  Moreover,  cognitive  research 
suggests  that  such  efforts  may  not 
make  a  sufficient  imprint  on  thinking 
to  affect  ongoing  analysis.  This  is 
because  information  that  is  inconsistent 
with  expectations  is  less  likely  to  be 
remembered  than  information  that  is 
consistent.  Since  alternative  thinking 
goes  against  the  grain  of  established 
thought,  its  ability  to  have  a  sustained 
impact  on  understanding,  individual  or 
collective,  is  always  open  to  question, 
all  the  more  so  if,  as  is  often  the  case, 
analytic  cadres  turn  over  fairly  rapidly.11 

A  continuous,  sustained  program  of 
small  to  medium-sized  efforts,  however, 
would  regularly  explore  different  possible 
outcomes  and  debate  assumptions,  all 
linked  to  incoming  information  about  the 
issue  under  consideration.  This  probably 
is  best  thought  of  as  an  ongoing 
conversation  (both  face-to-face  and 
electronic)  among  interested  parties, 
structured  to  encourage  divergent 


thinking.  Larger  efforts,  such  as  multiple 
scenario  workshops  or  multi-player 
games,  would  aim  at  feeding  results 
into  the  ongoing  dialogue,  not  simply 
publishing  and  moving  on.  Information 
technology  to  capture  and  automatically 
recall  both  previous  judgments  and 
alternatives,  perhaps  cued  by  keywords, 
would  be  essential  to  supplement 
human  memory,  further  stimulate 
debate,  and  provide  continuity  that 
survives  personnel  turnover. 


Alternative 
‘sense-making’ 
for  complex 
issues  would 
...  be  more 
freewheeling  and 
creative. 


Creative:  Traditional  alternative  analysis 
is  a  fairly  formal  process  with  some 
elements  of  creativity  but  with  a  strong 
emphasis  on  logical  argument  to  come 
to  clear  conclusions.  Alternative  “sense¬ 
making”  for  complex  issues  would, 
by  contrast,  be  more  freewheeling 
and  creative.  In  part,  firm  conclusions 
are  not  desirable  given  the  higher 
levels  of  uncertainty  inherent  in  these 
issues.  In  addition,  the  objective  is 
to  stimulate  pattern  recognition — to 
connect  the  dots — a  creative  process 
in  itself.  Cognitive  research  suggests 
that  judgments  about  the  likelihood  of 
events  often  reflect  the  “availability”  and 
vividness  of  memories  about  similar 
types  of  events.12  To  the  extent  that 
alternative  sense-making  can  help  to 
suspend  premature  judgment  and  make 
an  array  of  possibilities  come  alive,  it 
may  head  off  “failures  of  imagination”  by 
stimulating  exploration  of  alternative  dot 
arrangements. 

Increasing  creativity  within  the 
intelligence  field  to  enhance  intuitive 
judgment  can  be  accomplished  in 
a  number  of  ways.  Press  reports 
indicate  that  the  CIA  has  worked, 
for  example,  with  the  film  industry  to 
create  audio-visual  games  to  help 
analysts  “think  like  terrorists.”  Such 
major  efforts  would  have  a  more  lasting 


”See  see  Ziva  Kunda,  Social  Cognition:  Making  Sense  of  People,  (Cambridge:  MIT  Press,  2001),  pp.  161-210. 
12Kunda,  pp.  89-101. 
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Ideas  most 
often  ‘pop 
out’  of .. .  the 
‘contemplative’ 
mode  of 
thought,  which 
is  slow  moving 
and  largely 
unconscious . . . 


A  state  of  mental 
relaxation  is 
conducive  to 
the  playful 
workings  of  this 
contemplative 
mode . . . 


To  promote 
sustained 
awareness 
of  possible 
flaws  in  one’s 
thinking  requires 
a  continuous 
dialogue  among 
individuals 
with  different 
perspectives. 


impact  if  they  were  supplemented  by 
smaller,  more  frequent  efforts  geared 
toward  cultivating  creativity  at  both 
the  individual  and  organizational 
levels.  Systematic  use  of  the  wide 
array  of  divergent  thinking  techniques 
as  a  precursor  to  the  preparation  of 
intelligence  products  would  help  to 
put  a  wide  range  of  possibilities  on 
the  table  before  provisional  intellectual 
closure  was  reached.  Subject  matter 
specialists  would  be  paired  with  analytic 
methodologists  with  expertise  in  the 
application  of  such  divergent  thinking 
techniques.13 

A  conscious  mixing  of  mental  biases 
would  also  help  to  assure  that  different 
types  of  “eyes"  are  searching  for 
threatening  patterns.  The  specific 
suggestion  raised  in  workshop 
discussions  involved  the  practice — 
known  as  “barbelling” — used  by  some 
Wall  Street  firms,  which  involves  pairing 
young  financial  professionals  with  those 
over  50  to  marry  adventurousness  with 
caution  born  of  experience.  Firms  with 
such  an  age  distribution,  according  to 
a  noted  presenter  from  the  financial 
community,  tend  to  perform  better  than 
those  whose  professionals  cluster  in 
the  thirty-to-fifty  age  range.  Recruiting 
and  assigning  of  analysts  also  could 
be  designed  to  produce  teams  that 
consciously  mix  biases,  such  as 
perceptual  and  judgmental  thinking 
styles  (as  identified  in  Meyers-Briggs 
type  tests). 

Providing  analysts  with  much  greater 
time  and  freedom  to  think  about 
problems  than  is  normally  allowed 
by  hectic  intelligence  production 
schedule  could  also  foster  creativity. 


Ideas  most  often  “pop  out”  of  what 
cognitive  psychologist  Guy  Claxton 
has  described  as  the  “contemplative” 
mode  of  thought,  which  is  slow  moving 
and  largely  unconscious,  rather  from 
the  “deliberative”  mode,  which  is  fast 
moving  and  conscious.  A  state  of  mental 
relaxation  is  conducive  to  the  playful 
workings  of  this  contemplative  mode, 
something  hard  to  achieve  in  the  midst 
of  a  grueling  production  schedule.  Other 
options  include  enabling  analysts  to 
pursue  independent  projects,  contacts, 
and  conversations  seemingly  peripheral 
to  the  main  tasks  at  hand,  as  well 
as  occasional  working  retreats — for 
physical  setting  is  related  to  creativity  as 
well.14 

Collaborative:  Although  many  forms 
of  traditional  alternative  analysis  are 
collaborative,  an  individual  analyst  can 
do  others,  like  devil’s  advocacy  or  “what- 
if”  analysis.  Such  individual  efforts  are 
possible  because  the  aim  is  to  develop 
or  to  challenge  logical  arguments, 
something  that  the  individual  mind  does 
well.  However,  examining  alternatives 
on  a  sustained  basis  for  transnational 
issues  (and  for  comparably  complex 
and  dynamic  traditional  issues)  involves 
a  team  effort  embracing  a  variety  of 
disciplines  and  regional  specialties — far 
beyond  the  capabilities  of  a  single 
analyst.  Moreover,  introspection  is 
extremely  difficult  for  most  individuals, 
as  a  leading  cognitive  psychologist 
noted  at  one  workshop  and  is  amply 
documented  in  psychological  literature. 
To  promote  sustained  awareness  of 
possible  flaws  in  one’s  thinking  requires 
a  continuous  dialogue  among  individuals 
with  different  perspectives. 


1 3For  a  discussion  of  creativity  in  an  organizational  context  see  John  Kao,  Jamming:  Art  and  Discipline  of 
Corporate  Creativity ,  (New  York:  Harper  Business,  1997). 

14Claxton,  pp.  201-226.  Also  see  Judith  H.  Heerwagen,  “Creativity”  at  http://www.science.doe.aov  (last  visited 
13  July  2004). 
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One  collaborative  technique  discussed 
at  length  at  the  workshops  was  “out 
loud  sense-making" — a  structured 
process  for  orally  reviewing  assumptions 
and  alternatives.  Such  a  process  is  used 
by  surgical  teams  at  a  New  England 
hospital  in  response  to  recurrent  errors 
by  anesthesiologists  in  diagnosing 
respiratory  reactions — often  fatal — to 
anesthesia.  Now,  when  confronted  with 
breathing  problems,  the  team — including 
non-physicians — systematically  reviews 
possible  causes  out  loud  before 
arriving  at  a  conclusion,  dramatically 
reducing  the  error  rate.  Such  processes, 
enhanced  by  skilled  custom  design  and 
facilitation,  could  be  regularly  employed 
in  the  intelligence  arena  whenever 
anomalous  data  emerged. 

Another  collaborative  technique 
identified  at  the  workshops  involves  the 
greater  use  of  web-logs  (or  “blogs”)  as  a 
platform  for  intelligence  production.  US 
embassies  already  use  this  art  form — 
which  allows  for  continuous  posting  of 
information — as  their  primary  means  for 
conveying  information.  Unlike  published 
papers,  intelligence  web-logs  would 
be  a  form  of  “unfinished”  production  in 
which  both  intuitions  and  more  formal 
arguments  could  be  posted,  and  then 
challenged  by  those  with  alternative 
opinions.  Indeed,  web-logs  could  be  the 
forum  for  a  facilitated  virtual  dialogue — 
the  electronic  equivalent  of  out  loud 
sense-making. 

Counter-intuitive:  Intelligence  analysts 
need  to  focus  on  patterns  that  are 
different,  even  contradictory  to  those 
they  expect,  to  lessen  the  degree  to 
which  their  mental  models  inhibit  their 
ability  to  perceive  new  information. 

And  they  must  do  so  fairly  regularly 
to  promote  continuous  awareness  of 
possibilities.  This  is  not  easy  given  the 


natural  human  tendency — especially 
under  time  pressure — to  revert  to 
established  ways  of  viewing  the  world. 

One  possible  approach  is  to 
institutionalize  brief,  informal  exercises 
in  which  analysts  regularly  focus  on  how 
they  could  be  wrong,  along  the  lines  of 
what  Klein  calls  a  “pre-mortem.”15  In 
this  group  exercise,  individuals  imagine 
fiascoes  relating  to  their  areas  of 
responsibility — in  the  case  of  intelligence 
analysis,  a  warning  failure — and 
brainstorm  about  how  they  might  come 
about  and  how  they  could  be  headed 
off.  The  very  pointed  emphasis  on  the 
negative,  Klein  argues,  helps  to  shake  up 
complacency  that  arises  from  common 
overconfidence  in  judgments.  Such  an 
exercise  would  be  a  functional  equivalent 
of  the  “low  probability/high  impact” 
exercise  of  traditional  alternative  analysis, 
with  the  difference  that  its  informal  nature 
lends  it  to  more  frequent  use. 

Workshop  participants  also 
discussed  changing  the  culture  within 
intelligence  organizations  to  one  that 
is  more  conducive  to  self-questioning. 
Assessment  of  analytic  performance  often 
is  done  by  outside  bodies  with  negative 
findings  exploited  by  media  and  external 
critics,  thus  prompting  a  defensive 
stance  among  intelligence  professionals. 
There  has  been  only  modest  emphasis 
internally  on  looking  at  failures — and 
even  less  on  examining  successes — with 
an  eye  to  drawing  lessons  for  self- 
improvement,  notwithstanding  Sherman 
Kent’s  exhortation  for  such  introspection 
over  40  years  ago. 

The  systematic  and  non-punitive 
approach  of  the  nuclear  power  industry 
to  collecting  information  on,  and  learning 
from,  errors — associated  with  its  low 
failure  rate — provides  one  approach 


15Cited  above,  pp.  88-91 . 
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. . .  periodic 
‘After  Analysis 
Reviews’ . . . 
might  help  to 
create  a  more 
introspective 
environment. 


. . .  more 
experiential, 
interactive 
formats . . .  might 
better  capture 
the  attention 
and  imagination 
of  intended 
audiences  and 
strengthen 
retention  of 
insights. 


worth  considering.  Another  example 
is  provided  by  the  US  Army’s  Center 
for  Army  Lessons  Learned  (CALL), 
which  continuously  looks  at  operational 
problems  in  order  to  draw  implications 
for  improving  performance.  A  dedicated 
effort  to  continually  take  a  friendly  look 
at  getting  it  both  right  and  wrong  through 
periodic  “After  Analysis  Reviews” — 
modeled  on  the  Army’s  After  Action 
Review — might  help  to  create  a  more 
introspective  environment.16  Allowing 
analysts  the  time  for  such  exercises — or 
mandating  them — would  powerfully 
demonstrate  organizational  commitment 
to  learning. 

Consumer-friendly:  One  of  the  greatest 
challenges  of  any  “alternative”  effort  is 
to  effectively  communicate  the  message 
to  those  who  occupy  decision-making 
roles.  Decision-makers  are  buried  by  both 
information  and  tasks.  Motivating  them  to 
spend  time  reading  sophisticated  analysis 
in  general,  let  alone  analysis  that  queries 
existing  analytic  lines,  is  a  considerable 
challenge.  Several  presenters  at  the 
workshops  who  had  been  senior  officials 
in  the  terrorism  area  stressed  the  extent 
to  which  information  overload  had  grown 
in  the  post-September  11  environment. 
Moreover,  in  the  transnational  domain, 
many  potential  key  consumers  are  in 
middle  and  lower  operational  levels,  or 
outside  the  government,  and  thus  have 
even  fewer  contexts  for  understanding 
intelligence  information.  And,  of  course, 
in  order  to  carry  out  “alternative  sense¬ 
making,”  alternative  points  of  view  would 
need  to  be  put  before  this  wide  array 
of  harried  individuals  on  a  fairly  regular 
basis. 

One  way  to  accomplish  this  is  to  rethink 
the  concept  of  the  intelligence  “product.” 
Intelligence  organizations  continue  to 
insist  upon  written  prose  and  formal 
briefing  as  the  “gold  standard”  for 


disseminating  information  even  though 
adults  rarely  retain  more  than  ten  percent 
of  what  they  are  “told”  either  orally  or  in 
written  form.  Instead,  more  experiential, 
interactive  formats,  as  discussed  at  the 
workshops,  might  better  capture  the 
attention  and  imagination  of  intended 
audiences  and  strengthen  retention  of 
insights. 

•  Use  of  web-logs  would  give 
consumers — particularly  non¬ 
senior  consumers  without 
formal  feedback  processes — the 
opportunity  to  tap  in  from  time  to 
time  on  debates  within  the  analytic 
community  and  to  pose  questions 
themselves. 

•  RapiSims — rapid  simulations 
enabled  by  increasingly 
sophisticated  spreadsheet-based 
programs — would  allow  consumers 
to  manipulate  variables  to  generate 
alternative  outcomes.  Decision¬ 
makers  could  quickly  and  easily 
explore  a  range  of  possibilities 

in  a  way  that  is  more  likely  to  be 
retained  than  if  presented  in  a  long 
and  dry  formal  tome. 

•  Half  day  “gaming”  sessions — 
intentionally  kept  brief  to  allow  even 
the  most  harried  to  participate  on 
occasion — could  help  decision¬ 
makers  experience,  at  a  minimum, 
the  uncertainties  surrounding  an 
issue. 

Another  avenue  would  be  to  try  to 
strengthen  personal  relationships, 
such  as  through  increased  face-to-face 
contacts,  between  individual  analysts 
and  consumers  in  order  to  facilitate 
informal  exchanges  on  alternative 
outcomes.  One  presenter,  a  specialist 


16For  a  discussion  of  the  Army’s  after  action  process,  see  Nancy  M.  Dixon,  Common  Knowledge:  How 
Companies  Thrive  By  Sharing  What  They  Know ,  (Boston,  Harvard  Business  School  Press,  2000),  pp.  37-38. 
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in  corporate  innovation,  argued  that  the 
real  product  of  the  analytic  organization 
is  the  “analyst  rather  than  the  analysis,” 
just  as  professionals  in  other  knowledge 
industries — management  consulting, 
law,  etc. — rather  than  specific  products, 
are  the  key  selling  points  for  their 
organizations.17  A  professional  known 
by  and  enjoying  the  trust  of  consumers 
is  in  a  far  better  position  to  influence 
their  thinking  than  is  any  specific  report 


or  briefing.  Such  a  professional, 
if  inculcated  in  alternative  sense-making 
values  and  processes,  also  would 
be  in  a  position  to  periodically  draw 
consumers’  attention  to  the  results  of 
internal  exercises  on  assumptions  and 
outcomes. 

Some  of  the  practical  ideas  stemming 
from  workshop  discussions  are 
summarized  in  the  following  table. 


Key  Practical  Ideas 


Idea 

Implementation  and  purpose 

Develop  information  technology  to  store  and 
automatically  recover  hypotheses,  ideas. 

Aid  analysts’  memory  and  creative 
thinking,  and  promote  collaboration 

Employ  analytic  methodologists  with  training 
in  creativity  and  facilitation. 

Design  and  facilitate  divergent  thinking 
exercises  and  structured  dialogues 
aimed  at  surfacing  alternative  views. 

Consciously  mix  biases  in  teams  (e.g. 
“barbelling"). 

Increase  likelihood  of  alternative 
interpretations  of  evidence. 

Introduce  “out  loud  collaborative  sense¬ 
making”  processes. 

Structured  dialogues  to  consider  all 
possibilities. 

Use  web-logs  as  a  production  vehicle. 

Common,  continuous  platform  for 
carrying  out  a  “virtual  dialogue”  on 
alternatives. 

Regularly  do  after-action  reports 

Look  at  failures  and  successes  with  an 
eye  to  drawing  constructive  lessons 

Develop  information  technology  to  store  and 
automatically  recover  hypotheses,  ideas. 

Aid  analysts’  memory  and  creative 
thinking,  and  promote  collaboration 

Provide  opportunities  for  experiential 
learning  by  intelligence  consumers 

Brief  simulations/games  to  help 
consumers  comprehend  range  of 
uncertainty 

Promote  analyst  reflection  and  introspection 

Allow  time  off-line  for  pre-mortems  and 
after-action  exercises. 

'  Former  CIA  Deputy  Director  for  Intelligence  Winston  Wiley  often  said  that  the  Directorate  of  Intelligence 
produces  two  products:  intelligence  analysis  and  intelligence  analysts. 
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..  .the  need 
for  a  process 
to  produce 
sustained 
mindfulness  will 
only  grow. 


Throughout  the 
organization 
there  must  be  a 
‘culture’. . . 
that  values 
continuous, 
collective 
introspection. 


Conclusion 

Understanding  complex  transnational 
issues  requires  an  alternative  analysis 
approach  that  is  more  an  ongoing 
organizational  process  aimed  at 
promoting  continuous  “mindfulness” 
than  a  set  of  tools  that  analysts 
are  encouraged  to  employ  when 
needed.  The  latter  approach  lays  the 
responsibility  for  alternative  analysis  too 
heavily  on  the  shoulders  of  individual 
analysts,  who  may  not  have  the 
incentives  or  time  to  question  their  own 
or  their  colleagues’  lines  of  thinking.  And 
in  the  case  of  highly  uncertain,  rapidly 
changing,  boundary-crossing  threats 
such  as  those  in  the  transnational 
arena,  such  questioning  can  really 
only  take  place  as  part  of  an  ongoing, 
organizationally-supported  collaborative 
effort,  given  the  wide  range  of  disciplines 
and  institutions  that  contribute  to 
shaping  the  ultimate  analytic  product. 

With  the  global  landscape  likely  to 
feature  a  growing  array  of  hard-to-track 
threats  in  the  years  ahead — as  a  result 
of  the  increasing  availability  of  technical 
know-how  and  soaring  interconnectivity, 
among  other  factors — the  need  for 
a  process  to  produce  sustained 
mindfulness  will  only  grow.  Even  in  the 
case  of  more  traditional  intelligence 
issues,  developing  and  enhancing 
organizational  processes  designed 
to  promote  alternative  analysis  might 
encourage  more  systematic  use  of  these 
tools. 

Thinking  of  alternative  analysis  as  an 
organizational  process  has  important 
implications  for  how  the  intelligence 
analytic  organization  is  managed.  It 
means  that  senior  intelligence  officials 
must  make  challenging  assumptions 
and  considering  alternative  possibilities 
a  high,  indeed  critical,  priority.  It  calls 
for  reward  systems  that  encourage 
analysts  to  think  about  how  they  could 


be  or  have  been  wrong.  Production 
schedules  and  staffing  requirements 
need  to  be  adjusted  to  allow  time  for 
analytic  reflection.  Throughout  the 
organization  there  must  be  a  “culture” — 
one  inculcated  by  exhortation,  training, 
and  example — that  values  continuous, 
collective  introspection. 

Rethinking  the  concept  of  alternative 
analysis  may  even  mean  rethinking 
how  the  intelligence  field  defines  itself. 
For  instance,  the  CIA’s  Directorate 
of  Intelligence  has  self-consciously 
modeled  itself  after  the  academic  realm 
and  to  a  lesser  extent  the  press,  where 
questions  of  accuracy — getting  the 
data  and  facts  right — are  of  paramount 
importance.  But  should  it  not  think  of 
itself  as  more  equivalent  to  an  aircraft 
carrier  or  nuclear  power  plant,  where 
data  and  facts  are  only  the  beginning, 
and  questions  of  failure  drive  every 
aspect  of  operations? 
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